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Abstract 

There has been significant research over the past two decades in developing new platforms 
for spiking neural computation. Current neural computers are primarily developed to mimick 
biology. They use neural networks which can be trained to perform specific tasks to mainly solve 


pattern recognition problems. These machines can do more than simulate biology, they allow us 



to re-think our current paradigm of computation. The ultimate goal is to develop brain inspired 
general purpose computation architectures that can breach the current bottleneck introduced by 
the Von Neumann architecture. This work proposes a new framework for such a machine. We 
show that the use of neuron like units with precise timing representation, synaptic diversity, and 
temporal delays allows us to set a complete, scalable compact computation framework. The 
presented framework provides both linear and non linear operations, allowing us to represent 
and solve any function. We show usability in solving real use cases from simple differential 
equations to sets of non-linear differential equations leading to chaotic attractors. 


1 Introduction 


More than 50 years after the first Von Neumann single processor, it is becoming more and more 
evident that this sequential power greedy architecture scales poorly to multiprocessors. Despite 
the increase of the size of on-chip cache to stay away from RAM and to put the data closer to the 
processor, major processor manufacturers have run out of solutions to increase performances. 
The current solutions to use multicore devices and hyperthreading tries to overcome the problem 
by allowing programs to run in parallel. This parallelism is however limited as hyper-threaded 
CPUs even if they include extra registers still have only one essential element of most basic 


CPU features (Sutter 2005). 


The quest for a more power efficient alternative has seen a major breakthrough these last years, 
specially in asynchronous brain like dataflow architectures. Recent endeavours, such as the 
SyNAPSE DARPA program, led to the development of silicon neuromorphic neural chip tech¬ 
nology that allows to build a new kind of computer with similar function, and architecture to the 
brain. The advantage of these systems is then - power efficiency and the scaling of performance 
with the number of neurons and synapses used. There are currently several available platforms, 
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to cite the most sucessful: IBM TrueNorth (Merolla et al. 


2014), Neurogrid (Benjamin et al. 


[ 2014| ), SpiNNaker ( [Furber et al. 2014 1 , FACETS ( Schemmel et al. | 2008). These machines 
seem to be primarly intended to simulate biology, their main application being currently in the 
field of machine learning and more specifically running deep learning architectures such as deep 
neural networks, convolutional deep neural networks, deep belief networks and recurrent neu¬ 
ral networks. These techniques have shown to be efficient in several fields such as machine 
vision, speech recognition and natural language processing. Other options exists such as the 
Neural Engineering Framework (NEF), that has shown to be able to simulate brain functional¬ 


ities and provides networks that can accomplish visual, cognitive, and motor tasks (Eliasmith 


et al. 2012 1 . NEF intrinsically uses a spike rate-encoded information paradigm and a represen¬ 


tation of functions using weighted spiking basis funtions; it thus requires a very large number 
of neurons to compute simple functions. Other methods synthetize spiking neural networks 
for computation using Winner-Take-All (WTA) networks ( Indiveri[ 20011, or more vision dedi¬ 


cated spiking structures such as convolutional neural networks (Zamarreo-Ramos et al. 20131. 
Linear Solutions of Higher Dimensional Interlayers networks (Tapson & van Schaik 2012 1 are 
another class of approaches that are currently used to derive what is called Extreme Learning 
Machine (ELM) ( jHuang & Chen 20081. Recently the Synaptic Kernel Inverse Method (SKIM) 
framework (Tapson et al. 2013) has been introduced, it uses multiple synapses to create the re¬ 
quired higher dimensionality for learning time sequences. These methods use random nonlinear 


projections into higher dimensional spaces (Rahimi & Recht 2009; Saxe et al. 2011). They 
create randomly initialized static weights to connect the input layer to the hidden layer, and then 
use nonlinear neurons in the hidden layer (which in the case of NEF are usually leaky integrate- 
and-fire neurons, with a high degree of variability in their population). The linear output layer 
allows for easy solution of the hidden-to-output layer weights; in NEF this is computed in a 
single step by pseudoinversion, using singular value decomposition. 
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Our method goes beyond the classical point of view that neurons transmit information ex¬ 
clusively via modulations of their mean firing rates (Shadlen & Newsome 1998| Mazurek & 


Shadlen 2002| Litvak et al. 2003). There seems to be growing evidence that neurons can gen¬ 


erate spike-timing patterns with millisecond temporal precision in (Sejnowski, 1995 Mainen 


& Sejnowski, 1995; Lindsey et al. 

1997 

Prut et al. 

1998, 

Villa et al. 

1999; Chang et al. 


, 2000 Tetko & Villa 2001). Converging evidence suggests also that neurons in early stages 
of sensory processing in primary cortical areas (including vision and other modalities) use the 
millisecond precise time of neural responses to carry information (Berry et al. , 1997[ Reinagel 


& Reid 2000 Buracas et al. 1998[ Mazer et al. 2002 Blanche et al. | 2008 1. 


Our approach will also make use of precisely timed transmission delays. The propagation delay 
between any individual pair of neurons is known to be precise and reproducible with a sub¬ 
millisecond precision ( Swadlowj 1985[ 1994 1 . Axonal conduction delays in the mammalian 
neocortex (Swadlow. 1985 1988 1992 1 are known to range from 0.1 ms to 44 ms. Finally 
we will also use the property of biological neurons that states the same presynaptic axon can 
give rise to synapses with different properties, depending on the type of the postsynaptic target 
neuron (Thomson et al. 1993, Reyes et al. 1998 Markram et al. 19981. 

In this paper we are interested in deriving a new paradigm for computation using neuron 
like units and precise timing. The goal is to design micro neural circuits operating in the pre¬ 
cise timing domain to perform mathematic operations. We show that when using computation 
units that have common properties with biological neurons such as precise timing, transmission 
delays, and synaptic diversity, it becomes possible to derive a Turing complete framework that 
can compute every known mathematical function using a non Von Neumann architecture. The 
presented framework allows to derive all mathematical operators whether they are linear or non 
linear. It also allows relational operations that are essential to develop algorithms. The precise 
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timing framework has a compact representation and it uses a low number of neurons to solve 
complex equations. Examples will be shown on first and second order differential equations. 
The developed methodology is in adequation with scalable neuromoiphic architectures that 
make no distinction between memory and computation. Every synapse of each computational 
unit of the model simultaneously stores information and uses this information for computation. 
This contrasts with conventional computers that separate memory and processing thus causing 
the von Neumann bottleneck where most of the computation time is spent in moving information 


between storage and the central processing unit rather than operating on it (Backus 1978). The 


developed approach is easily scalable and is designed to naturally operate using an event-driven 
massive parallel communication similar to biological neural networks. 

The next section describes the neural model used in this work and the encoding scheme 
chosen to represent values in the exact timing of spikes. It also describes neural networks 
implementing elementary operations that can be assembled to implement arbitrary calculus. We 
then present results of applications of such networks, followed by a discussion on the methods 
proposed in this work and a conclusion. 

2 Methods 

2.1 Neural model 


These neuron-like computational units use the following neural model: 


= 9e + gate.g f 
0 


dffe 

dt 


r, - 

'/• dt ~ 


-9f 


( 1 ) 


V is the membrane potential of the neuron. We consider here that there is no leakage of 
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the membrane potential (or that the time constant of this leakage is much slower than all the 


other time constants considered in this work, in which case it can be neglected). g e represents 
a constant input current which can only be changed by synaptic events, gj represents input 
synapses with exponential dynamics. These synapses arc gated by the gate signal which is 
triggered by synaptic events. For the experiments presented in this work, we use r,„ = 100s and 
Tf = 20ms. 

We thus distinguish 4 type of synapses, where w is the weight of the synapse: 

• V — synapses directly modify the membrane potential value: V <— V + w, 

• g e — synapses directly modify the constant input current: g e 4— g e + w , 

• gg — synapses directly modify exponential input current: .(// •(— g f + w, 

• gate — synapses : w = 1 activate the exponential synapses by setting gate 4— 1; w = — 1 
deactivate the exponential synapses by setting gate <— 0. 

All synaptic connections are also defined by a propagation delay between the source and target 
neurons. 

A neuron spikes when its membrane potential reaches a threshold, i.e.: 


V>V t (2) 

it then emits a spike and is reset by putting back its state variables to: 

V <— K-eset (3) 

9e 0 (4) 

9f <- 0 (5) 

gate 4— 0 (6) 
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without loss of generality, we will consider Vt = lOmV and F reS et = 0 to simplify the following 
equations. 

In the following subsections, we use the following notations. T syn is the propagation delay 
between two neurons for standard synapses. T neu is the time needed by a neuron to emit a spike 
when triggered by an input synaptic event; it can model, for instance, timesteps of a neural 
simulator. In the experiments presented in this work, we use T syn = 1ms and T neu = lOus. We 
define w e as the minimum excitatory weight for V — synapses required to trigger a neuron in 
its reset state, and Wi the inhibitory weight of counteracting effect: 


W e = V t 

(V) 

Wi = -W e 

(8) 


Standard weights for g e — synapses will be defined in the next subsection. 

2.2 Signal representation 

The main idea of the method proposed in this work is to represent values as the precise time 
interval in between two spikes. 

If the series e n (i) is the list of times at which neuron n emitted spikes, with i the index of 
the spike in the series, neuron n encodes the signal u(t) by: 

u(e n (i)) = f~ 1 (e n (i + 1) - e n (i)), Vi = 2 .p, p € IN, (9) 

with i an even number and / _1 the inverse of the encoding function / of our choice. 

The encoding function / : IR. —> R can be chosen depending on the considered signals in 
a particular' system and adapted to the required precision. / computes the interspike time At 
associated with a particular value. In the following work, we chose to represent values using the 
following lineal' encoding function: 

At = f{x) = T min + x.T cod , (10) 
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with x G [0,1] and T co d the elementary time step. 

This representation allows us to encode any value between a minimum and a maximum 
interspike (of T nnn and T max = (T m i n + T co d)). We chose to use a minimum interspike to 
encode zero for several reasons. If the two spikes encoding a value originate from one unique 
neuron or are received by a single neuron, this minimum interspike gives time to recover from 
the first spike before spiking again. T m i n allows networks to react to the first input spike and 
propagate a state change before the second encoding spike is received. In the experiments 
presented in this work, we use T nnn = 10ms and T coc j = 100ms. 

We could also choose a logarithmic function to allow encoding a large range of values with 
dynamic precision (precision would be smaller for large values). 

To represent signed values, we use two different pathways for the two different signs. Posi¬ 
tive values will be encoded by causing a neuron to spike and negative values by eliciting another 
neuron to spike. We arbitrary chose to represent zero as a positive value. 

To ease the understanding and the routing of several networks, each implementing simple 
operations, we add some interface neurons to these networks. For instance, the input of the 
networks are materialized by special input neurons. Their output by some output, output + or 
output - neurons. Other special neurons used for interaction between networks are also marked. 
In all the following figures describing networks, neurons in blue will be input neurons to the 
networks whereas neurons in red will be output neurons of the circuit. 

The simple network presented in Fig. [T] encodes a constant value. It shows the design prin¬ 
ciples which will be used in the rest of this section. In the network shown in Fig. [T] the recall 
neuron is an input. When a spike is received by recall, the constant value encoded in the net¬ 
work is output to the output neuron. In this example, the output is generated by two different 


synaptic connections from recall to output. They generate two output spikes with the interspike 


(IjTsyn) 



Figure 1: A network generating a constant value when required. When the recall neuron is 
activated, the output neuron will generate a pair of spikes coding for value x. 

corresponding to the encoded value. 


We define standard weights for g e — synapses. Let w; acc be the weight value for g e synapses 
to cause a neuron to spike from its reset state after time T max = T nmi +T ca( ]. According to Eq.[l] 
we have: 

v t = — .r max (ii) 

such that: 

™acc = V t .^~ (12) 

•L max 

We also define weight tD acc as the g e value necessary to have a neuron spiking from its reset 
state after time T co j. The same equation gives us: 


w a 


= v,.. 


Tcod 


(13) 


We can now describe different neural networks implementing elementary operations such as 
: memories, synchronizer, linear combination or non-linearities such as multiplications, directly 
operating on inter spike intervals. 

2.3 Storing data: memory 
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Figure 2: Inverting Memory: this network stores a value encoded in a pair of input spikes x 
(to the input neuron) by integrating current on the g e dynamics of neuron acc during the input 
interspike. The value, stored in the membrane potential of the acc neuron is read out when the 
recall neuron is triggered. Releasing a pair of spikes with neuron output corresponding to 1 — x. 
Blue, red and black neurons are, respectively, input, output and internal neurons. 

Inverting Memory Fig.[2]presents an Inverting Memory network. This network is constituted 
of two input neurons (in blue): input and recall , one output neuron (in red): output and 3 internal 
neurons (in black): first, last and acc. Details and proof of the inner working of the network 
can be found in Appendix |A.1| The chronogram of spikes during operation of this network is 
presented in Fig. [3] When a pair of spikes arrives at the input neuron, they are sorted by the. first 
and last neurons. Their synaptic connections are such that first will only spike in response to 
the first encoding spike of the pair (at time f^) and last will only spike in response to the second 
encoding spike of the pair (at time tf n ), thus seperating the two input spikes. 

first and last are then respectively starting and stopping the integration of the membrane 
potential of neuron acc at times tl t and t^ nd such that the value of ace’s membrane potential 
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V, 



Figure 3: Inverting Memory: chronogram of network operation for an input at times t\ n and 
t? n and a recall at time tj ecall . (Input spikes are drawn in blue, output spikes in red. Green plots 
show the membrane potential of interesting neurons.) 

after the second input spike is, with ATj n = tf Q — t l ui the input interspike: 

7/1 

Vsto = — .(AT in -T min ) (14) 

T m 

When the recall neuron is triggered, the integration starts again until reaching Vt such that we 
get an output interspike A T out = tf iui — t ] nut following: 

V t = ^A.(A ^ - T min ) + ^.A T out (15) 

I'm I'm 

Considering the definition of m acc , we have Vi-T rn /iv ;iCC = T rnax , such that 

A T out = T m&x - (A T in ~ T min ) (16) 

We thus obtain an output spike corresponding to the maximum temporal representation of a 
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value (T max ) minus the actual coding time (AT in — T m i n ) of the input value. Chaining two of 
these Inverting Memory networks, can store and recall a value without modification. 

We can also notice that the value is stored in the inter spike timing needed to represent 
the value and can be recalled as soon as the value has been completely fed into the Inverting 
Memory network. 



Figure 4: Memory: this network stores a value stored in the inter spike timing of two input 
spikes (to the input neuron) similarly to the Inverting Memory network. Two accumulators, acc 
and acc2 are used to output on the output neuron the same value as previously received on the 
input neuron. Blue, red and black neurons are, respectively, input, output and internal neurons. 


Memory Fig. [4] presents a Memory network. Detailed explanations and a chronogram of the 


network’s operations can be found in Appendix A.2 This network is based on the Inverting 
Memory network introduced in the previous paragraph, 2 accumulator neurons acc and acc2 
are added to invert the stored value twice. If we follow the same reasoning as in the previous 
paragraph, acc spikes T max after the first encoding spike is received and acc2 starts integrat¬ 
ing when the second encoding spike is received. Because acc stops acc2 's integration process, 
the value stored in acc2 ’s membrane potential after acc spikes is, with AT rn the input inter¬ 
spike (we present here a simplifyed result to ease the notations, the full result is available in 
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Appendix | A. 2[): 


uu 

Vsto= — .(T max - AT in ) (17) 

An 

Note that the time at which acc2 ends its integration happens after the input value has been 
completely fed into the Memory network. This is the reason why we added the ready neuron 
which is triggered when the input value has been stored in the Memory network and is ready to 
be read out. We then have the output interspike A T out following: 

V t = —.(T max - A Tin) + —-A T out (18) 

I'm I'm 

such that 

A T out = ATj n (19) 



Figure 5: Signed Memory: this network is using a Memory network to store a signed value and 
its sign. Depending on the sign of the input (i.e. if it is received on the input + or input - neuron), 
the wrong output neuron (output + if the input is negative) is inhibited so that the output of the 
internal Memory network is directed to the output corresponding to the input sign. Blue, red 
and black neurons are, respectively, input, output and internal neurons. 


13 














Signed Memory Fig.[5]presents a Signed Memory network. This network uses a Memory net¬ 
work to store a value and a small state machine, implemented by neurons ready + and ready -, 
to store the sign of the input. Detailed equations and the chronogram of the network’s opera¬ 
tions can be found in Appendix |A.3| The internal Memory network is linked in parallel to the 
positive and negative pathways of the Signed Memory. When an input is fed into one of these 
two pathways, only the corresponding ready neuron receives some excitation. When the recall 
neuron is triggered, only the ready neuron corresponding to the sign of the input spikes (because 
of the excitation contributed by the stored input). The ready neuron will then inhibit the wrong 
output such that only the output neuron corresponding to the correct sign will fire. 

Synchronizer Fig. [6] presents a Synchronizer network. This network receives N different 
inputs and synchronizes their first encoding spikes on the output end. Detailed equations and 
the chronogram of the network’s operations can be found in Appendix |A.4| It is implemented 
using N Memory networks. The sync neuron keeps track of the number of received inputs. 
When all the N inputs have been received, this neuron spikes, starting the readout process of 
the different memories at the same time, thus synchronizing all the outputs. 

Furthermore, the same principle can be used with Signed Memory networks to obtain a 
Signed Synchronizer network. 

2.4 Relational operations 

Minimum Fig. [7] presents a Minimum network. This network implements the minimum op¬ 
eration on 2 inputs. It outputs the smallest of its 2 inputs as well as an indicator of which input 
was the smallest one. If the two inputs, inputl and input2, arc synchronized and because our 
encoding function is increasing with its input value, the minimum of the 2 inputs is the one for 
which the second encoding spikes arrives first. This is what the smaller 1 and smaller2 neurons 
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delay T S y n 



Figure 6: Synchronizer: this network is synchronizing a set of input values so that the first 
spikes encoding every value on its output side happen at the exact same time. It can be used to 
resynchronize values before networks requiring this condition. It uses a set of Memory networks 
to store the different input values and a sync neuron which recalls all these stored values as soon 
as the last one of them has been stored. Blue, red and black neurons are, respectively, input, 
output and internal neurons. 

are extracting. This information is also used to inhibit the excitatory contribution of the largest 
input to the output neuron in order to output only the smallest of the input values. Detailed proof 
and the chronogram of operations can be found in Appendix |B.1| 

Maximum Fig. [8] implements the maximum operation on 2 inputs. This networks follows 
the same principle as in the Minimum network, it differs by inverting the detection relation: 
when the first input is the smallest, it triggers larger2 because the second input must then be the 
largest. The drive of the output neuron is simpler in this case as the inputs are synchronized. 
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Figure 7: Minimum: this network outputs the smallest of its two inputs, which have to be 
synchronized, and an indicator signal on neuron smallerl or smaller2 corresponding to which 
input has been considered as the smallest one. Because we chose an encoding function / which 
output increases with its input, the smallest input corresponds to the one for which the second 
encoding spike arrives first. This is what the output neuron is extracting. Blue, red and black 
neurons are, respectively, input, output and internal neurons. This network only contains V — 
synapses 

The maximum value corresponds to the one for which the second encoding spike is the latest. 
Detailed proof and the chronogram of operations can be found in Appendix |B. 21 

2.5 Linear operations 

Subtractor Fig. [9] presents a Subtractor network. It is here presented in its simplest form. It 
will be expanded in a second stage. This network computes the difference between inputl and 
input2 and directs the output depending on its sign either to the output + or output - neuron. 
If the two inputs are synchronized, the difference between the two is directly given by the 
interspike between the two second encoding spikes. This is the information syncl and sync2 


16 









Figure 8: Maximum: this network outputs the largest of its two inputs, which have to be 
synchronized, and an indicator signal on neuron larger 1 or larger2 corresponding to which 
input has been considered as the largest one. Because we chose an encoding function / which 
output increases with its input, the largest input corresponds to the one for which the second 
encoding spike arrives last. This is what the output neuron is extracting. Blue, red and black 
neurons are, respectively, input, output and internal neurons. 

neurons are extracting. It also implements the same idea as in the Minimum network (see Fig. [7]) 
to compute the sign of the output. When the output sign is known, syncl or sync2 inhibits the 
pathway to the wrong output neuron such that the output spikes are directed to the correct one. 
The detailed proof and the chronogram of operations can be found in Appendix |C.1| 

A more robust version of the Subtractor network is presented in Fig. [10] The networks 
adds the zero neuron and its connections (in magenta in the figure). In the previous ’simple’ 
implementation shown in Fig. [9} when both inputs are equals, the two parallel pathways of 
the network are triggered at the same time and the lateral inhibition has no time to select a 
winning pathway. In that case, the output is emitted both on output + and output - which can be 
problematic for the following networks expecting only one of the two pathways to be activated. 
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Figure 9: Subtractor (simple): this network substracts its two inputs, which have to be synchro¬ 
nized. The resulting value is directed to two different neurons, output + or output -, depending 
on its sign. Because the two inputs are synchronized, the difference is directly given by the 
interspike between the second spikes of both inputs. The sign is determined using the same idea 
as in the Minimum network (see Fig. [7]). Blue, red and black neurons are, respectively, input, 
output and internal neurons. This network only contains V — synapses 

To solve this problem, we add the zero neuron with a set of fast synaptic connections. They 
allow to detect the case of equality between the two inputs. In this case, the output - pathway is 
quickly inhibited to produce spikes coding for the zero output only on the output + neuron. 


Linear Combination Fig. [IT] presents a Linear Combination network. It computes the linear 
combination of N signed inputs with arbitrary coefficients op, —otN-i '■ 

N -1 

S = ^2 Oii-Xi, (20) 

i=0 
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Figure 10: Subtractor (full): this nework implements the same operation as the one presented 
in Fig. [9] The simple version requires that the input are not equal so that the inhibition resulting 
from the sign determination has time to propagate. The added zero neuron and its connections 
(in magenta) detects this particular case and drives the network to output a zero on neuron 
output + if the two inputs are equal. Blue, red and black neurons are, respectively, input, output 
and internal neurons. This network only contains V — synapses 

where xi , i e 0,.... A r — 1 arc the different inputs of the network. It uses the same principle 
as in the Memory network to store values in an accumulator. To implement the coefficients of 
the sum, we multiply the synaptic weight of the accumulation current by the coefficient corre¬ 
sponding to the input. To handle the signs of the inputs and coefficients, we use 2 accumulators. 
The first one is storing intermediate results which are positives (i.e. when the sign of the input 
is the same as the one of its associated coefficient) while the second stores negative values (i.e. 
when the sign of the input is different from the one of its associated coefficient). When all the 
inputs have been fed into the network, the sync network is triggered, causing the readout process 
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Figure 11: Linear Combination: this network computes the linear combination of a set of 
inputs with given coefficients ao,... ,a^-i : s = a i- x i- The network is accumulating 

positive and negative sub-operations (i.e. oti.Xi) in 2 distinct accumulators. Synaptic connec¬ 
tions in the network are shown for ao > 0 and a \- < 0. The two resulting values are then read 
out by the sync neuron when all inputs are known, synchronized and substracted from one an¬ 
other to obtain the full, signed, result s. An indicator neuron start is provided to notify when the 
result is ready. Blue, red and black neurons are, respectively, input, output and internal neurons. 

of these accumulators. Their content are inverted as for the Memory network and then synced 
before entering a Subtractor network. This last network computes the difference between the 
positive and negative contributions of the different inputs and produces a signed output. A start 
neuron is then triggered to spike to indicate that the computation has ended. This signal can be 
used to trigger further networks. All details and proofs can be found in Appendix |C.2| 
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2.6 Non-linear operations 



Figure 12: Log: this network computes the natural logarithm of its input by using the gf dynam¬ 
ics of neuron acc. The input value is first stored in the membrane potential of the acc neuron. 
When the second encoding spike arrives, a gj — synapse is used to obtain a delay function 
of the log of the current value of acc ’s membrane potential. Blue, red and black neurons are, 
respectively, input, output and internal neurons. 

Natural Logarithm Fig. [12] presents a Log network capable of computing the natural log¬ 
arithm of its input value by exploiting the dynamics of a gg — synapse. Detailed proof and 
the chronogram of operations can be found in Appendix |D.1| When an input value is fed into 
the input neuron, this value is first stored in the membrane potential of acc by integrating with 
weight m acc between the spike of first and last. Because the spike from first is delayed by an 
additional T m ; n compared to the one from last, the value stored in acc ’s membrane potential is, 
with A Ti n = Tnim + A T co d the input interspike: 

T T W & CC / Am m \ ^acc » m A T cod 

v — - 1 min ) — - ■t+J-cod — Ft-—- (21) 

T~m T~m * cod 

When this integration process is stopped by last’s spike, a synaptic event is also triggered on a 
gg — synapse of acc with weight () mu it (another synaptic event also enables the gj dynamics by 
activating the gate state), acc ’s membrane potential thus follows the evolution given by solving 
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the differential system Eq. 0: 


V = V t . 


ATcod 

T cod 


+ 9'muit~(l — e t ^ T f) 
I'm 


( 22 ) 


If we chose g mu i t such that: 


We obtain 


_ T r T m 
flmult *t- 

T f 


v = Vt '^L + Vt '( l-e-t/V) 


T C od 

Considering neuron acc will then spike at time t s where V = Vt, we get: 

V t = V t 


Tcod 


A Tr 


Tcod 


cod = e~ ts/Tf 


(23) 


(24) 


l - + V t .(l e~ u ' r f) 

(25) 


(26) 

( AT cod \ 

(27) 

\ T cod ) 


which is a positive value because A T co d < T co d by definition. Adding the delays of the synaptic 
connections to the output neuron, we get an output interspike A T out : 


AT out — T m [ n + Tf. log 


( T co d \ 

VA T cod ) 


(28) 


We thus obtain a network capable of generating an output proportional to the natural logarithm 
of its input. 


Exponential Fig. [represents the Exp network that computes the exponential of its input value 
by exploiting the dynamics of a 5/ — synapse. The detailed proof and the chronogram of op¬ 
erations can be found in Appendix |D.2| When an input value is fed into the input neuron, a 
synaptic event is triggered on a gf — synapse of the acc neuron with weight g mu it as defined 
in Eq.[23]by ncuron /mv/. acc ’s membrane potential thus follows the evolution given by solving 
the differential system Eq.[l]until last spikes: 

^ = ftnuit — (1 - e" Vr/ ) = Vt . (1 - e-^f ) (29) 

Tm 
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Figure 13: Exp: this network computes the exponential of its input by using the pj dynamics of 
neuron acc. A value function of the exponential of the input value is first stored in neuron ace’s 
membrane potential by gating a p/ — synapse to acc on the second encoding spike of the input. 
acc ’s membrane potential is then read out to obtain the result. Blue, red and black neurons are, 
respectively, input, output and internal neurons. 

This synaptic activity is then gated when neuron last is triggered and blocks arc’s membrane 
potential value to : 

V = V t .(l -e- AT ‘° d/r f), (30) 

with A Ti n = T m j n + AT co d the input interspike (Because of the additional delay of T nnn in 
first’s pathway in comparison to the one of last). The spiking of last also triggers a readout of 
acc’s membrane potential by initiating a g e synaptic event with weight u) acc such that acc spikes 
after time t s following: 


Vt 

- V t .(l e~ ATcod/r f)+ Wacc .t s 

(31) 


Tm 


Vt 

= V t .(l-e- AT ^ r f) + V t .^- 

cod 

(32) 

ts 

= T cod .e- AT ^ T f 

(33) 
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Adding the delays of the synaptic connections to the output neuron, we get an output interspike 


AT ou t: 


A Tout = T m in + T cod .e~ AT ^/ T i 


(34) 


We thus obtain a network capable of generating an output proportional to the exponential of its 
input. 


Going back to the Log network from the previous paragraph, its output was : 

= AT‘°f - Tmi n = Tf. log (^) (35) 

If this output is fed into an Exp network, we obtain the output: 

A T^f = Train+ T cod .e~ ATl °°y r f (36) 

= Train + T cod .e l °^ AT ^/T oll ) (37) 

= Tann + A T cod (38) 

The Exp network is thus capable of inverting the Log network. One can take advantage 
of that to implement several common non-linearities by applying simple operations in between 
the Log and Exp networks. For instance, summing two logarithms will allow multiplication, 
subtracting them will implement division, multiplying a logarithm by a constant will compute a 
power function, ... 


Multiplier Fig. [T4]presents a network we name Multiplier. The detailed proof and the chrono¬ 
gram of the network’s operations can be found in Appendix |A.3| This network is based on the 
principles used in the Log and Exp networks. The product s of the 2 inputs x\ and x '2 is obtained 
using the well known following equation: 

s = X 1 .X 2 = exp(logxi + logX 2 ). (39) 
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Figure 14: Multiplier: this network multiplies its two inputs. It computes the logs of its inputs 
through neurons accJogl and accJog2, sum them up and gets the exp of this value through 
neurons acc^exp to obtain the product: s = x±.X 2 = exp (log a;i + log £ 2 )- Blue, red and black 
neurons are, respectively, input, output and internal neurons. 

Neurons accJogl and accJogl 's membrane potentials are first loaded with the two inputs ( in- 
putl and input2). When the 2 inputs have been received, an exponential circuit is triggered 
through acc_exp. To obtain the product of the input, this circuit has to be stopped after a time 
corresponding to the sum of the natural logarithm of the 2 inputs. Because the absolute value 
of the natural logarithm can be larger than 1 for small inputs (i.e. it is larger than the maximum 
value representable by our encoding scheme), we cannot use a Linear Combination network to 
sum the logs. To overcome this problem, we sum these values by triggering the logarithm com¬ 
putation of the 2 inputs successively: the sync neuron, which is detecting the end of the second 
input, it activates at the same time the exponential circuit and the logarithm of the first input. 
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When the first input’s logarithm is output, it triggers the logarithm of the second input which, 


when computed, stops the exponential circuit. At that point in time, the accjexp neuron contains 
in its membrane the product of the 2 inputs. It is then read out to compute the actual output of 
the network. 



Figure 15: Signed Multiplier: this network computes the result and the sign of the multiplica¬ 
tion of its two inputs. A Multiplier network is used to compute the absolute value of the result. 
This value is then directed to the correct output (output + or output -) by a small truth-table 
implemented in neurons signl, sign2, sign3 and sign4 which are determining the sign of the 
output from the signs of the two inputs. Blue, red and black neurons are, respectively, input, 
output and internal neurons. 
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Signed Multiplier Fig. [13] presents a Signed Multiplier network. It computes the product of 
two inputs independently of their sign. In parallel, a set of neurons: signl, sign2, sign3 and 
sign4 are used as a truth table to determine the sign of the output from the sign of the inputs. 
When the output sign is known, the wrong output pathways (either output + or output -) is 
inhibited to direct the output of the Multiplier network to the right output neuron. The different 
sign neurons implement a truth table with the excitatory connections they receive from the input 
neurons. Input connections and weights are designed such that only one sign neurons spikes 
when an input is fed into the circuit. This “winning” neuron can then be associated to an output 
sign. Lateral inhibition between the sign neurons is present to suppress residual activations by 
the input of non-winning sign neurons (this allows all the sign neurons to go back to their reset 
state once the output sign is computed). 

2.7 Differential equations 

Integrator Fig. [16] presents a Integrator network that allows to reconstruct a signal from its 
derivative fed into its input. It is using a multiplier and an accumulation network with a Linear 
Combination network. The output of this accumulator network is looped into its first input with 
a unit gain. The input, composed of input + and input -, is fed into this accumulator with a 
gain dt corresponding to the chosen integration timestep. Each time an output is produced on 
output + and output -, the indicator neuron new Input is triggered to notify that the integrator 
is ready to receive its next input. This system is thus driven by its input: every time an input is 
provided, the corresponding output is computed. Two auxiliary input neurons are also provided. 
The init neuron loads the integrator with its initial value. This allows the internal state of the 
integrator (through the Linear Combination network) to be set to an initialization value. The 
start neuron feeds a zero into the input of the integrator, thus computing its first output. 
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Figure 16: Integrator: this network integrates an input over time. The init and start neurons 
are used to initialize the inner state of the integrator. The network is then driven by its input. 
Everytime an input is received, it is integrated and added to the latest value of the output. When 
a new value is output, the new .input neuron spikes, requesting for a new input. Blue, red and 
black neurons are, respectively, input, output and internal neurons. 

System design All the networks presented in this section can be assembled to achieve more 
complex computational tasks. Multiplier and Linear Combination networks can be associated 
to compute arbitrary functions on some state variables. Integrator networks can then be used to 
solve systems of differential equations. Examples of such network will be demonstrated in the 
next section. 


3 Results 

We implement in this section different computational tasks. We start by implementing linear dif¬ 
ferential equations with a first order and a second order system. In a second stage, we implement 
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a more complex set of non-linear differential equations from Edward Lorenz. 


3.1 Linear differential equations 



Figure 17: Neural network implementing a first order differential equation. It is composed of a 
Constant network providing the input, a Linear Combination network computing the derivative 
of X and an Integrator network computing X from dX/dt. 

First order system We first implement a first order differential system. This system solves 
the following equation: 

dX 

T -~jjr + X(t) = Xqo (40) 

We implement this network as shown in Fig. [Pausing 3 of the networks described in the previous 
section: 

• a Constant network is providing the input X^ to the system, 

• a Linear Combination network is computing dX/dt, 

• an Integrator network is computing X from its derivative. 


29 























(a) Different values of r (b) Different values of ATo 

Figure 18: Neural implementation of a first order filter. Blue curves show the input values to the 
filter, red curves show the output values of the filter for different parameters. (Horizontal axis is 
time, every circle correspond to an actual output encoded by spikes of the network.) 


The in it and start neurons enable to initialize and start the integration process, init has to be 
triggered before the integration process can take place to load the initial value of the Integrator 
network, start has to be triggered to output the first value from the Integrator network. When 
an output is provided by the Integrator network, the Constant network is activated using the 
new Jnput neuron of the Integrator. Hence feeding two values into the Linear Combination 
computing the new derivative of the output. This derivative is then integrated by the Integrator 
to obtain a new output. With the implementations presented in the previous section, this network 
requires 118 neurons. Results of its simulation with different set of parameters for r and 


and for dt = 0.5 are presented Fig. 18 


Second order system To add complexity, we now implement a second order differential sys¬ 
tem. This system solves the following equation: 


Jl_ d 2 X 

Wq dt 2 


+ 


£ dX 

cuq dt 


+ X(t) = X oo 


(41) 
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Figure 19: Neural network implementing a second order differential equation. It is composed of 
a Constant network providing the input, a Linear Combination network computing the second 
derivative of X and two Integrator networks computing dX/dt and X from d 2 X/dt 2 . (Signs of 
the different signals have been omitted to increase readability.) 



(a) Different values of u>o (b) Different values of £ 

Figure 20: Neural implementation of a second order filter. Blue curves show the input values 
to the filter, red curves show the output values of the filter for different parameters. (Horizontal 
axis is time, every circle correspond to an actual output encoded by spikes of the network.) 

We implement this network as shown in Fig.[l9]using 4 of the networks described in the previous 
section: 

• a Constant network is providing the input X <*, to the system, 


31 


















• a Linear Combination network is computing , 

• a first Integrator network is computing dX/dt from the second derivative of X, 

• a second Integrator network is computing X from its first derivative. 


With the implementations presented in the previous section, this network requires 187 neurons. 
Results of its simulation with different set of parameters for £ and cuq and for dt = 0.2 are 


presented Fig. 20 


3.2 Lorenz attractor 
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Figure 21: Neural network implementing Edward Lorenz’s non-linear differential equation sys¬ 
tem. (Signs of the different signals have been omitted to increase readability.) 


We now implement the set of non-linear differential equation proposed by Lorenz (1963): 


dX 

dt 

= a(Y(t) — X(t)) 

(42) 

d Y 

dt 

= pX(t)-Y(t)-X(t).Z(t) 

(43) 

dZ 

dt 

= X(t).Y(t) - 0Z(t) 

(44) 
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Figure 22: Neural implementation of a set of non-linear differential equations. These equations 
converge to the Lorenz attractor. The plot shows the evolution of the system in the phase space, 
every axis being one of the state variables X, Y, Z. Fixed points of the system are shown in 
blue, the origin point is shown in green. Every circle corresponds to an actual output encoded 
by spikes of the network. 

using cr = 10,/3 = 8/3 and p = 28 to ensure chaotic behavior of the system. We also use a 
variable substitution to obtain state variables X , Y and Z evolving in [0,1] so that they can be 
represented by our framework. The initial state of the system is set to X = —0.15, Y = —0.20 
and Z = 0.20. We use an integral step of dt = 0.01. 

We implement this network as shown Fig. [21] by using 9 of the networks described in the 
previous section: 

• 2 Signed Multiplier networks compute the non-linarities contained in Y and Z ’s deriva¬ 
tives, 

• 3 Lineal - Combination networks compute the derivatives of X, Y and Z, 
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• a Signed Synchronizer network allows to wait for the three derivatives to be computed 


before evolving the system’s state, 

• 3 Integrator networks compute the new state from the derivatives of X, Y and Z. 


With the implementations presented in the previous section, this network requires 549 neurons. 


Results of its simulation are presented Fig. 22 We can observe that the system is behaving as 


expected, following the strange attractor described by Lorenz. 


4 Discussion 

In this work, we choose a linear encoding function to map values into inter spikes. In this case, 
it results in a direct trade-off between the time needed to represent a value and the time precision 
of the system. A finer time precision leads to a larger number of possible different values in a 
given maximum representation time. In the paper we chose to set time scales that are compatible 
with neuroscience evidence. However, current hardware allows much faster time scales up to 
nanoseconds. In that case not only can we obtain higher precision but also faster computation 
times. 


The number of neurons used in the current implementation of the examples given in this 
paper can be reduced in size. They have been designed to ease comprehension and to be easy 
to plug into each other. In almost all the networks, the first and second encoding spikes for the 
output are generated by different neurons of the network. All the networks have a pan - of first 
and last neurons which task is to separate the two incoming spikes. By directly routing these 
two spikes independently, the whole implementation would be much more efficient and would 
require much less neurons. Less neurons would also imply less spikes and therefore less energy 
requirements and less latency in signal propagation. 
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For feedforward architectures, the different layers of computation could also be pipelined. 
This means that if a task is composed of a series of operations which can be considered as layers, 
the full operation would require data to go through all the layers, with all layers being active 
only a fraction of the time. But at every point in time, one layer could also be computing its op¬ 
eration on a different input, such that all of the layers are always active. In the end, one can get 
one output per pipeline period, even if the whole computation takes several pipeline period to 
be completed (as many pipeline period as there are steps in the computation, the pipeline period 
being the longest time needed by one stage to output its result). This means better throughput 
for feedforward computation, but imposes a minimum delay on feedback (because the result to 
be fed back is not available before a certain number of pipeline periods). 


The developed framework can be used to compute any algorithm, it is intersting to notice 
that it is naturally compatible with all type of time oriented AER (Address Event Representa¬ 


tion ([Boahen 2000)) data and all kind of AER sensors (Dclbruc k et al. |2010| ). The use of 
interspike makes the framework particularly adapted to process luminance time encoded events 


data from the neuromorphic camera ATIS (’’Asynchronous Time-based Image Sensor”) (Posch 


et al. 2011 2008). Thus every event-based machine vision algorithm developed so far could 


be systematically implemented on neural boards. 


A question outside the scope of this paper relates to the architecture of the platform that 
should implement this computation paradigm. Several solutions could be possible. A pure ana¬ 
log chip could be used. Analog design is known to be a difficult endeavour, moreover we would 
need to robustify the framework to overcome mismatch. Several options are possible, the most 


straightforward is to use regularization techniques such as the one introduced in (Benosman 
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et al. 2014) or simply using calibration techniques to provide a precisely timed output from 
analog chip similar to what has been introduced in (Sheik et al. . 2012( 2011) . 

A mixed-signal integrated circuit mixing both analog circuits and digital circuits could also be 
considered. Exponential decays could be generated using analog circuits while the remaining 
operations could use digital circuits. Finally the last solution could be a pure digital architecture 
preserving the principles introduced in the paper such as inter spike encoding, local compu¬ 
tations that allow to overcome the Von Neumann bottelneck. In this case, we could use local 
binary computation units to perform conventional computation rather than using membrane po¬ 
tentials, synapses, delay,.... Everything being local to the units, the architecture would still be 
efficient. This by far is the less elegant solution, but it could be a rapid and an intermediate step 
toward a full true neuromorphic implementation. 


Conclusion 

This work introduced a new clockless framework to build a multipurpose neuromorphic com¬ 
puter. Instead of representing values as a set of bits in a register or a part of some central memory, 
values are coded in the precise timing between events happening in the system. This dataflow 
framework we called STICK (Spike Time Interval Computational Kernel) offers a new method 
to design computing platforms where memory and computation are intertwined. By removing 
the numerous accesses to a central memory, inherent to standard computers, we free ourselves 
from the Von Neumann bottleneck. The systems scales naturally, the more neurons are available 
the more computation can be performed. Building a large machine consisting of several small 
elementary computational units allows to build natively massively parallel machines which rely 
on neuromorphic engineering principles to reduce their power consumption: energy is needed 
to produce events which only happens when information is produced. 
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Moreover, the STICK framework offers a way to design large neural networks that accom¬ 


plish complex tasks by design of their architecture instead of only exploiting the randomness of 
a given connectivity topology. It also defines all the weights necessary to obtain a functional 
solution without the need for a costly (in terms of energy, time or computational power) learning 
process. A hardware fabric consisting of a large number of the proposed computational units 
with dynamic connection could then be used and adapted to successively solve different tasks. 
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Appendix 

This appendix presents the detailed proofs of all the networks described in Section[2] 

A Storing data: analog memories 

A.l Inverting Memory 

The Inverting Memory network (see Fig. [ 2 ]) receives 2 spikes on the input neuron at times tj n 
and tf n such that AT rn = tf n — tl n encodes its input value. When input spikes at t ] m , synaptic 
connections are activated towards neurons first and last. Because of the synaptic delays and 
weights, last’s membrane potential is set to Vt/2 and first spikes at time f’ irst = t. ] m + T syn + 
T neu , T neu being the time needed by first to produce a spike. When first spikes, an inhibitory 
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Figure 23: Inverting Memory: chronogram of the network for an input at times t\ n and tf n and 
a recall at time fj ccall - (Input spikes are drawn in blue, output spikes in red. Green plots show 
the membrane potential of interesting neurons, recall of Fig. [3]) 

connection to itself sets its potential to — V) while a second connection triggers the integration 
of neuron acc after a delay T syn + T m i n with weight u; acc . We then have: 


t 


i 

St 


+i 

''first 


+ ?syn + T n 


(45) 


— i* n + 2.T syn + T neu + T m j n 


(46) 


When input spikes for the second time at time tf n , first’s membrane potential gets back to its 
reset value while last reaches its threshold. This produces a spike from neuron last at time 
^last = tfn + ^syn + T neu . The connection to acc with delay T syn and weight —w acc stops acc ’s 
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integration at time: 


t 


i 

end 


+1 

‘last 


+ T S 


syn 


(47) 


— ^in 4" 2 .Tf. 


syn 


+ T n , 


(48) 


During this integration window we had, for neuron acc, g e = w :iCC such that, the membrane 
potential of acc at time is: 


Vsto = —-(tL-tst) 

(49) 

T m 


= — .(tf n -(4 + T min )) 

(50) 

T~m 


— Wacc (AT. T . ) 

(51) 


Tm 


When the recall neuron receives an input at time t J ecall , acc’s integration starts again at time 
& = tj ccall + T syn . At the same time, its second connection triggers a spike of the output neuron 
at time: 


t 


t 

out 


^recall + (2-7"syn + ^neu) + ^n. 


(52) 


= t 


t 

recall 


+ 2.T syn + 2 ,T n , 


(53) 


The integration stops again when acc reaches its threshold at time giving the following 
equation: 


V t = 


• (^end 4t) 4" 


^end — (^recall 4"-^syn) (ATj n T m ; n ) + V). 


W a 


By definition of m acc , we have V).r,„/u; acc = r max , so : 


(54) 

(55) 


^end — ^recall 4" ^syn 4" T max (/\Ti n T m ; n ), 


(56) 


Because accl then needs the time T neu 

t 


to produce a spike, we get: 


1 _ j-2 , 

acc ~~ ‘'end > i neu 


(57) 
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We thus get the second spike of output at time: 


f 2 

''out 

— ^accl + ^syn + T nevi 

(58) 

t 2 

‘'out 

= ^recall + 2.T syn + 2.T neu + T max — (A Tj, n — T m i n ) 

(59) 

such that: 

AT out = f out — t out 

(60) 


= T max - (A T in - r min ). 

(61) 

A.2 Memory 



The Memory network (see Fig. [ 4 } receives 2 spikes on the input neuron at times i ; ’ n and t 2 n 

such that ATi n = t 2 n — tj n encodes its input value. With the reasoning used in the previous 

subsection, we get: 


^first — ^in 4“ ^syn 4" T neu 

(62) 


^last = ^in 4“ ^syn 4" T neu . 

(63) 


When first spikes, it triggers integration in acc’s membrane potential. Because of the synaptic 
delay, we get: 


4t — ^first + T syn — t- m + 2.T syn + T Iieu 


(64) 


Neuron acc continues to integrate its w aC c input until reaching its threshold. This gives us the 
following equation: 

V t = ^.(Cd - 4) (65) 
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Figure 24: Memory: chronogram of the network for an input at times tj n and tf n and a recall 
at time Recall- (I n P ut spikes are drawn in blue, output spikes in red. Green plots show the 
membrane potential of interesting neuron.) 


by definition of u> acc , we thus get: 


•'end 


i — 


A 

" , end2 


fgt + Imax 

tl n + Imax + 2.T syn + T neu 
^end 9" 2~neu 

= + Imax + 2.T syn + 2.T neu 

— t 1 4 -T 

= + Imax + 3.T syn + 2.T neu 


( 66 ) 

(67) 

( 68 ) 

(69) 

(70) 

(71) 
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At the same time, integration stalls in acc2 's membrane potential after last spikes, which corre¬ 


sponds to: 


4t2 — ^last + Tsyn — t in + 2.T SV n + T r 


syn i -'■neu 


The membrane potential of acc2 after the end of the first integration phase is thus: 

Vsto = ^■(t 1 end2 -t 1 st2 ) 

" • (^max + fhi — tin + ^syn + ^neu) 

■{T max — A Ti n + T syn + T neu ) 


Tm 
^acc 
Tin 
^acc 


T, 


m 


When the recall neuron is triggered, it stalls acc2 's integration again at time: 


^st2 — ^recall 4~ ^syn 


Which then produces the first output spike at time: 


^out — ^recall 4~ Tsyn + ^neu- 


This second integration phase of acc2 finishes when its threshold is reached: 


= 


w 


acc /,2 


(*end2 ~ t st2 ) + V 8 to 


^max — tend2 ^recall -^syn T T max A T xn + ^syn T T n 


tend ~ ^recall + A Tj n Tneu 


We thus get a spike from acc2 at time: 


^acc2 ^end2 4~ Tneu ^recall 4" ATj, 


leading to the second output spike at time: 


^out — Fcc 2 + Tsyn + Tneu — ^recall + A Tj n + T syn + T x 


L syn ' J-neu 


such that: 


A Tout = tL, -1, 


out ‘"out 


= AT, 


(72) 

(73) 

(74) 

(75) 

(76) 

(77) 

(78) 

(79) 

(80) 

(81) 

(82) 

(83) 

(84) 
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A.3 Signed Memory 




Figure 25: Signed Memory: chronogram of the network for an input at times t\ n and tf n and a 
recall at time fj ecall . (Input spikes are drawn in blue, output spikes in red. Green plots show the 
membrane potential of interesting neurons), (a) Depicts the case of a positive input whereas (b) 
depicts the case of a negative input. 

The Signed Memory network (see Fig. [ 5 ]) receives 2 spikes on the input neuron at times t- m 
and tf n such that A7)„ = tf n — t- m encodes its input value and the receiving neuron encodes 
the sign of the input (input + for positive inputs, input - for negative inputs). Let’s consider, 
without loss of generality, the case where the input is positive (Fig. [25} a)). For each of the 2 
input spikes, the ready + neuron receives a synaptic contribution of 0.25//; fi . When the input has 
been completely fed into the network, ready +’s membrane potential is thus resting at a value 
of Vt/2 while ready -’s is still resting at its reset potential. At the same time, the input spikes 
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are fed into the central Memory network (see previous subsection) independently of its sign. 


When the value is stored in the Memory network, it outputs a spike on its Rdy output, which is 
propagated to the ready neuron. 

When the recall neuron is triggered, its connections contribute to ready + and ready -’s 
membrane potentials with a weight of 0.5w e . This leads to a spike of the ready + neuron at time 
fg ign while ready -’s membrane potential moves to V//2. The lateral inhibition between the 2 
ready neurons then sets ready - back to its reset potential. When ready + spikes, it triggers the 
recall of the Memory network and at the same time inhibits the output neuron corresponding to 
a negative value, output -, setting its potential to —2.V t . When the Memory network outputs its 
stored value, spikes are transmitted to both output. Because of their respective potential at this 
moment, only the positive output output + spikes, while output -’s membrane potential is set 
back to its reset potential by the 2 output spikes of the Memory network. 

Fig. (25jb) shows the same principle applied to a negative input. One can notice that the 
ready + and ready - neurons are implementing a small state machine routing the spikes produced 
by the central Memory network to different output neurons depending on the input neurons. 

A.4 Synchronizer 

The Synchronizer network (see Fig. [6]) for N inputs uses N Memory networks. Fig.[26]presents 
the chronogram of this network for N = 2. Every time an internal memory has finished storing 
an input, its Rdy output activates the sync neuron with a weight w e /N. Thus, after i inputs have 
been presented, sync’s membrane potential V s 2 ync is: 

V: ync = ^.Vf (85) 

The sync neuron thus spikes after the N th and last memory is ready. It then recalls the values 
stored in all Memory networks at the same time, effectively synchronizing the first encoding 
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Figure 26: Synchronizer: chronogram of the network for inputs at times tf n0 and tf n] . 
(Input spikes are drawn in blue, output spikes in red. Green plots show the membrane potential 
of interesting neuron.) 

spikes of all its outputs. 


B Relational operations 


B.l Minimum 


The Minimum network (see Fig. [7]) receives 2 different inputs (a pair of spikes) from each input 
neurons inpull (fj L nl and t? nl ) and input2 (t\ n2 and tf n2 ) such that ATj n i = tf n] — f i 1 nl and 
ATj n 2 = t\ r a ~ ij ' n 2 encode its 2 inputs. Let us consider, without loss of generality, the case 


where the first input (input 1) is smaller than the second one as shown in Fig. 27 Assuming the 
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Figure 27: Minimum: chronogram of the network for inputs at times t ,- ml , tf n] and tj n2 , tf n2 . 
(Input spikes are drawn in blue, output spikes in red. Green plots show the membrane potential 
of interesting neuron.) 

inputs to be synchronized, we have: 

tlni = tin 2, (86) 


When inputl and input2 are triggered by the first encoding spike of each input, smaller 1 and 
smaller2’s membrane potentials are set to V//2 and the output neuron spikes after a delay due 
to the connection from inputl and input2 such that: 


fouti — iini + 2.T syn + 2.T neu . 


(87) 


When the smallest input (in this case inputl ) receives its second encoding spike at time t 
smallerl neuron reaches its threshold and emits a spike at time: 

^smallerl = ^inl + ^syn + 2~neu- 


2 

ini) 


the 


( 88 ) 


The spike from smallerl inhibits the other input (here input2) such that it will not be triggered 
by its second encoding spike (as can be seen in the chronogram). It also inhibits the smaller2 
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neuron such that its membrane potential goes back to its reset potential. The second encoding 
spike from inputl and the spike from smaller 1 are, in addition, enough contribution to trigger a 
second spike in the output neuron at time: 


t 


out 


= mm - 


lin + 2.r S y n + Tneuj ^smaller 1 “1“ -^syn} -^neu 

= min { tf n j + 2.T syn + T neu , t'( n \ + 2.T ayn + T neu } + T n , 


— ^inl + 2.T syn + 2.T neu . 


We thus get an output pah - of spikes spaced in time such that: 


AT ou t — t out t out — /p nl t inl — A Tj : 


ini • 


(89) 

(90) 

(91) 


(92) 


The output is thus corresponding to the smallest of the 2 inputs of the network while the indicator 
provides which of the two inputs is the smallest ( smaller 1 ). The same reasonning can be applied 
to the case where the second input {input!) is the smallest. 


B.2 Maximum 

The Maximum network (see Fig. [8]) receives 2 different inputs (a pair of spikes) from each input 
neurons inputl (t l inl and tf n ,) and input2 (t\ n2 and tf n2 ) such that AT, n \ = tf nl — f i 1 nl and 
A7) n 2 = tf n2 — f* n2 encode its 2 inputs. Let’s consider, without loss of generality, the case 


where the second input ( input2) is larger than the first one as depicted Fig. 28 The 2 inputs 
being synchronized as a prerequisite of the network, we have: 


t 1 - t 1 

''ini t 'in2 > 


(93) 


When inputl and input2 are triggered by the first encoding spike of each input, larger 1 and 
larger2’ s membrane potentials are set to 14/2 and the output neuron spikes after a delay due to 
the connection from inputl and input2 such that: 


f-outl — fin2 T T/ yri + T rK . u . 


(94) 
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Figure 28: Maximum: chronogram of the network for inputs at times fj nl , tf nl and t- m2 , tf n2 . 
(Input spikes are drawn in blue, output spikes in red. Green plots show the membrane potential 
of interesting neuron.) 

When the smallest input (in this case input 1) receives its second encoding spike at time tf n] , 
we know that the other input has to be the larger one. Hence, the larger2 neuron reaches its 
threshold and emits a spike at time: 


t 


1 

larger2 


— ^fnl + -^syn + T nexl 


(95) 


The spike from larger2 inhibits the larger 1 neuron such that its membrane potential goes down 
to —Vt/2, while the connection inputl to output raises output’s membrane potential to 14 / 2 . 
When the second encoding spike from input2 is received, the connection from input2 to larger 1 
moves back largerV s membrane potential to its reset potential while its connection to output 
triggers a spike at time: 


u out 


- t 2 

~ L in2 


+ Xsyn + T ni 


(96) 
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We thus get an output pair of spikes spaced in time such that: 


^■T ou t — f out t out — tf n 2 ^in2 — ATj n 2- (9V) 

The output is thus corresponding to the largest of the 2 inputs of the network.while the indicator 
provides which of the two inputs is the largest (larger 1 ). The same reasonning can be applied 
to the case where the first input (input 1) is the largest. 

C Linear operations 

C.l Subtractor 



Figure 29: Subtractor: chronogram of the network for inputs at times t ] mi , t? nl and t- in2 , tf n2 . 
(Input spikes are drawn in blue, output spikes in red. Green plots show the membrane potential 
of interesting neuron.) 
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The Subtractor network (see Fig. [9]) receives 2 different inputs (a pair of spikes) from each 
input neurons input 1 (^ nl and t? nl ) and input2 (t- m2 and tf n2 ) such that A7) n i = tf n] — t? nl and 


ATj n 2 = tf n 2 ~ t\n 2 enc °de its 2 inputs. Let us consider, without loss of generality, the case 


where the first input (input 1) is larger than the second one as depicted Fig. 29 Assuming the 
inputs to be synchronized, we have: 


4i = 42- (98) 

When inputl and input2 are triggered by the first encoding spikes of each input, they activate 
the syncl and sync2 neurons such that their membrane potentials are now set to V/ /2. When the 
second encoding spike of the smallest input (here, input2 ) is received, the activation from input2 
to sync2 is sufficient to trigger a spike at time: 


6 sync2 


— ^in2 + ^svn + ^1 


syn 


(99) 


This spike inhibits the inbl neuron after a time T syn , moving its membrane potential to — V) and, 
because the sign of the output is now known, it triggers and output spikes on output + at time: 


^out — 4ync2 3.T syn + 2.T neu + T neu — T^ n2 + 4.T SV n + 4.T n , 


s yn 


( 100 ) 


It also contributes to output -’s membrane potential with an activation of w e at time isync 2 + 
2min+3.T syn +2.T neu . But before this contribution reaches output -, the inb2 neuron is triggered 
and produces a spike at time: 


^inb2 — ^sync + T syn + T neu 


( 101 ) 


which inhibits output - with weight 2w t at time t\ nh2 + T syn = tl ync2 + T syn + T neu . This 
inhibition thus happens before the direct excitation from sync2, which then leads to output - not 
emitting a spike and its membrane potential to be set to — Vt after receiving these 2 spikes. 
When the second encoding spike of the largest input is received, the spike from inputl 


50 



triggers a spike from sync I at time: 


^syncl — ^inl 4" T syn + ^neu- 


( 102 ) 


This spike leads to the inhibition of inb2 to a membrane potential of — 14 and an excitation of 
inbl back to its reset potential. It also activates output + back to its reset potential and triggers 
an output spike at time: 


tout — tsynci + Train + 3.T syn + 2.T neu + T neu — ff nl + T m ; n + 4.T syn + 4.T neu (103) 


We thus get a positive output as expected and an output value: 


A T m ,± — 


t 2 -t 1 

"'out •'out 


T ■ 4-t 2 — t 2 
± min T- ml <'in2 


— Tiuin + (tjnl t inl ) (tj n 2 t; n2 ) 


— ^min + (ATj n i — ATj 


m2 ) 


— Tmia + (ATj n i — T m ; n ) — (A Tj n2 — T m i n ). 


(104) 

(105) 

(106) 

(107) 

(108) 


Knowing that for each value x, we encode it as a time interval f{x) = T mm + x.T C(l> \- we have: 


A Tout — T m in T X U i \. 7 r(1( ] Xi n 2 ■ T c()( ] 


— Tmin T {Xin 1 2'in2)'Tcod 


(109) 

(HO) 


which corresponds to the encoding of the result input 1 — inputs. 


C.2 Linear Combination 


The first part of the Linear Combination presented Fig. 11 can be decomposed in a series of 
simpler circuits to what has been presented before for the inverting memory. For each of the 
N inputs, one branch is managing positive inputs while the other one is managing negative 
inputs (only one of these 2 can be activated in any computation). The architecture of each 
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of these branches can be seen as an inverting memory (see Fig. [2]) storing a value either in 
accl+ or accl-. The targeted accumulator is chosen depending on the sign of the input and 
the sign of its associated coefficient at to represent the sign of the input’s contribution to the 
result. accl+ is accumulating all positive contributions (positive inputs with positive coefficients 
and negative inputs with negative inputs) and accl- is accumulating all negative contributions 
(negative inputs with positive coefficients and positive inputs with negative coefficients). With 
the same reasoning leading to Eq. ( [51] ) we can get the contribution of an input i to its associated 
accumulator, if A T- n is the input interspike associated to the considered input: 

II) 

V l sto = \ai\. — .(Ali n - T min ), (111) 

An 

The integration in the accumulators being linear, we obtain the membrane potential stored in the 
2 accumulators after all inputs have been fed into the network: 


-\raccl-\- _ 

* sto 


V 


accl— 

sto 


E 

vi to 



(112) 

i£l+ 





E 

i£l+ 

\oti • 

W'acc 

7~m 

.(A Tt-T min ) 

(113) 

E 

vl 



(114) 

iei~ 





E 

i£l- 

\ai . 

®acc 

An 

.(A Ti n -T min ) 

(115) 


where X + is the set of inputs contributing positively to the output and X~ is the set of inputs con¬ 
tributing negatively to the output. When the N inputs have been fed into the network, the sync 
neuron finally receives enough excitation to produce a spike at time tl ync . This spike triggers the 
readout process of accl -r and accl- and, at the same time, starts integrating in neurons acc2+ 
and accl-. This process is similar to the one used in the Memory network (see Appendix |A.2[ ). 
If we consider the positive accumulator, we obtain spikes from accl+ and acc2+ at time t^ ccl+ 
and facc 2 + respectively with the following conditions, where = ^ ync + T syn is the time at 
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which the integration begins: 



< 

Vt = ^f.(facc 1+ -4 1 t)+^o Cl+ 

= 3 ^ SL -(facc2+ — fgt) 

(116) 

By definition of m ac 

, we thus get: 




facc2+ — t s t 4” Tm ax 

(117) 

and 

T 

- 1 - max 

= faccl+ — ^st 4" y ^ \ a i\-(AT in — T m i n ) 

(118) 



i£l+ 



^accl+ 

= ^st 4“ Tmax ~ y ' \ a i\-{AT in — T m ; n ) 

(119) 



iex+ 



Neuron inter+ is thus producing 2 spikes with an interspike AT^ t such that: 

^'inter = ^acc2+ + T m in + ^syn + T neu (faccl+ + T syn + ?n eu ) (120) 


^ ] | a i\-(AT in 7mi n ) + T u 

tex+ 


( 121 ) 


The same reasoning on accl- and acc2- leads to a pair of spikes on neuron inter- with an inter- 
spike A Tr fer : 

ATinter = \ a i\-{AT in — T m ; n ) + T m j n (122) 

i£l~ 

This two values are then synchronized by a Synchronizer network described previously and 


subtracted from one another such that the output is, according to Eq. (107 1 : 


^ Tout — A T^ ter A T inter + T m i n (123) 

= \ a i\-{AT- n — Train) ~ \ a i\-(AT^ n ~ T m in) + T m in (124) 

i£l+ ieT- 

N—l 

= e i- a i-(AT^ n — T m i n ) + Train (125) 

i =0 

where e* is +1 if input i is positive and —1 otherwise. This is the expected result of the linear 
combination. 
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D Non-linear operations 

D.l Natural Logarithm 



Figure 30: Log: chronogram of the network for an input at times fj n and tf n . (Input spikes arc 
drawn in blue, output spikes in red. Green plots show the membrane potential of interesting 
neurons). 


The Log network (see Fig. 12 1 receives 2 spikes on the input neuron at times t ] m and tf n such 


that A Ti n = t'i n — t\ n encodes its input value. The same reasoning leads us to obtain the spike 
times of the first and last neurons at: 


^first — ^in 4~ ^syn + ^neu 

(126) 

^last = ^in + ^syn + T neu 

(127) 

The acc neuron is thus integrating from time: 


4t = ^first + ^syn + 2mi n 

(128) 

= t} n + 2.T syn + T m i n + T neu 

(129) 
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to time: 


^end — + T e 


last ' ± syn 


— t in + 2.T syn + T neu 


(130) 

(131) 


The membrane potential of neuron acc at the end of this integration phase is thus: 


Vsto = ^££ (t i nd _4) 

"(All ~ ^in — Anin) 
-(AT in - T min ) 


7~m 
^acc , 
7m 


(132) 

(133) 

(134) 


If we name A T co d = A7) n — T m ; n and considering the definition of u; acc , we have: 

A T co d 


Vsto = V t . 


Tcod 


(135) 


The other synaptic connections from last to acc also activate the gj dynamics of the acc neuron 
at time t^ nd . When the gj — synapse gets activated, acc ’s membrane potential follows the 
following evolution obtained by solving the differential system Eq. 0: 


V = V sto + g mn[t ^(l - e-^^nd )/Tf) 


(136) 


for t > tl nd . According to Eq. (23), we chose p m uit = Vt- — so that: 


T f 


V = V t . 


A Tcod 
T C od 


+ V t .(l - 


(137) 


The acc neuron will then spike at time t\ icc when the condition V = Vt is met. This gives us: 

ATcorf 


Ft = V t . 


Tcod 

— f Oacc And)/ 7 "/ 


_)_ p) . ^ 1 — £ Acc And )/ T /^ 


A T coc [ — t+i 

Tcod 


+ 1 _ ^ (^T co d\ ! + i 

Fee T/. log I — 1 T Fnd 


(138) 

(139) 

(140) 


The first output spike is generated by the connection from last to output , thus: 


Fut — ^last 4“ Tneu + 2.T syn 


(141) 
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While the second output spike is generated by the connection from acc to output , thus: 


^Ollt 


— ^acc + 2~neu + T syn + T m ; n 


— ~Tf. log ^ + tend + ^neu + T syn + 7mi n 

= ~ T f- 1°§ f j, “ ^ + tjast + 2~iieu + 2.T S y n + T n 


This gives us the output A T ou t. 


(142) 

(143) 

(144) 


A T out 


t 2 -1 1 

‘'out ''out 


f AT co d ^ 

-Tf. log I —- I + T min 


T min + r f . log 


f T c pd \ 

VA T C0 J 


(145) 

(146) 

(147) 


D.2 Exponential 



Figure 31: Exp: chronogram of the network for an input at times f* n and t 2 n . (Input spikes are 
drawn in blue, output spikes in red. Green plots show the membrane potential of interesting 
neurons). 
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The Exp network (see Fig. 131 receives 2 spikes on the input neuron at times tj n and t? such 


that A Ti n = tf n — t\ u encodes its input value. The same reasoning leads us to obtain the spike 
times of the first and last neurons at: 


A 

c first 


''last 


ti n + Tsyn + T neu 

(148) 

t[ n + T syn + T neu 

(149) 


The first neuron then triggers the gj- dynamics of neuron acc at time: 

*st = ^first + ^syn + Tmin = tin + T neu + 2.T syn + T m i n (150) 

Solving the differential system Eq. ([!]), arc’s membrane potential is following the evolution: 

V = 5muit —(l-e- (t -^ )/T 0 (151) 

Tm 

= V t .(l - e-V-** )/T f) (152) 

This evolution is stopped at tg nd by the connection from last to acc through its action on the 
gate signal: 

^end = ^last 5“ -^syn = ^in T Tneu T 2.T S y n (153) 

At the end of this phase, acc ’s membrane potential is thus equal to: 

Vsto = 


V t .(l - 

(154) 

Vt.(\ — e — (*?n — *hi — Tnin )/ T f^ 

(155) 

V t .(l — g— (AT in —T min )/r/\ 

(156) 

V t .(l - e~ ATc ° d/T f) 

(157) 


with A T co d = A Ti n ~ T miu . At the same time, last is starting a second integration process of a 
g e — synapse, arc’s membrane potential is then following the evolution: 

V = V sto + U ^.(t-t 1 end ) (158) 

Tm 
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This behavior leads to a spike at time t^ cc when the condition V = V ( is met: 


Vt 

_ T/ , W & cc (+1 A \ 

— Vsto-r T'-acc ''end/ 

Tm 

(159) 

Vt 

- V t .( 1 e ATc ° d/r f)+ .(tl cc fend) 

cod 

(160) 

t 1 

v acc 

= fend + T cod .e~ ATcod / Tf . 

(161) 

The first output spike is produced by the connection from last to output, thus: 



^out — ^last ^neu + 2.T syn 

(162) 

The second output spike is 

produced by the connection from accl to output, thus: 


+2 _ 
''out 

face 4" -^neu T ^syn T T m [ n 

(163) 

= 

^cod- e ATcod / r f + fg nd + T neu + T syn + T m i n 

(164) 

= 

2cod-C ^ T cod/ r f _|_ + T neu + 2.T syn + T m ; n 

(165) 


This gives us the output A T out : 

A T ou t = t out — f out (166) 

= T cod .e- AT ^ T f + T min (167) 


D.3 Multiplier 


The Multiplier network (see Fig. [14] ) receives 2 different inputs (a pair of spikes) from each 
input neurons input 1 (fl nl and tf n] ) and input2 (t l m2 and tf n2 ) such that A7) n i = tf ni — fl nl and 
ATj n 2 = t] n 2 ~^\n 2 enc °de its 2 inputs. The first layers of the network, composed of the neurons 
input, first, last and accJog for each inputs are similar to the Logarithm network. Considering 
the results from Appendix |D.1[ when the 2 inputs have been fed into the network, we get 2 
potentials stored in accJogl and acclogl (respectively V s t 0 \ and V s i 0 2 )' 


Vstoi = 

Vstrt = 


(168) 
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Figure 32: Multiplier: chronogram of the network for inputs at times t? nl , tf n] and t l m2 , tf n2 . 
(Input spikes are drawn in blue, output spikes in red. Green plots show the membrane potential 
of interesting neurons). 

When the 2 inputs have been fed into the network, spikes from last / and last2 activate the sync 
neuron which spikes at time tL nc . This triggers the readout of the log of input 1 in the accJogl 
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neuron at time: 


4tl — 4yncl + ^syn- 


(169) 


Results from Appendix |D.1| tell us that this neuron will thus spike at time: 


,1 i / ^ TcodJ \ ,1 

*io gl = ~ T f • lo e l Tcod ) +4t- 


(170) 


A spike from the accJogl neuron will then trigger the readout of the log value of input2 in the 
accJogl neuron at time: 

^st2 = ^logl + ^syn- (171) 


accJog2 will then produce a spike at time: 


ti og 2 = ~ T f • log + t 2 st . 


(172) 


Which will trigger the first output spike at time: 


^out — ^log2 "6 2.T SV n- 


L syn- 


(173) 


At the same time, the sync neuron also started the gj dynamics of neuron cicc_exp at time: 


4t3 — 4ync + 3.7s yn . 


(174) 


This process is stopped by the spike from neuron accJog2 at time: 


t 


i 

end3 


= t 


1 

l°g2 


+ T S 


syn 


(175) 


Results from Appendix |D.2|tell us that the potential stored in acc_exp at that time is thus: 


V st o3= V t .(l -e (t end3 4 3 )A/) 


(176) 


and that the integration process started by the second connection from accJog2 to acc exp 
(acting on g e ) will result in a spike at time: 


t 


i 

exp 


= t 


1 

end3 


+ T cod .e (^ndS 4 3 )A/. 


(177) 
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This will result in the second output spike at time: 


^out — ^exp + T syn + T m in- 


(178) 


The output of the network is thus the interspike A T out such that: 


AT™,* = tL, -1. 


t 


out ‘'out 

1 


— Tinin + ^exp ^log2 ^syn 

= T min + T cod .e-(*end3-43)/^ + ^ - T syn 
= T min + Tcod.e-^endS-^ts)/-/ 


(179) 

(180) 
(181) 
(182) 


From previous equations, we get: 


A - t 1 
*'end3 ‘stS 


— ^log2 ^sync 2.T E 


L syn 


Which then gives us: 


( ^T co d2 \ ,2 _ ,1 _ r. rri 


Tf. log ^ T i I ° st: ‘sync 

{ AT cod2 \ 


syn 


_l_ +1 _ +1 _ T 

i Mop-I (/ svnr. - 1 - S 1 


= ~Tf. log rjf ^ J T q og i ''sync -‘syn 

, (AT cod2 \ , / AT co di\ 

= ~V • log - - T f . log —- 

/ AT„„ji .AT„jo \ 

= Tf * log 


^cod 


A Tout = Tmin + AT cod i.AT cod2 


(183) 

(184) 

(185) 

(186) 
(187) 


(188) 


which corresponds to the encoded value of the produce of the value encoded by the 2 inputs. 
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